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Abstract 

It is common in college courses to test students on the required readings for that course. 
With a rise in online education it is often the case that students are required to provide 
evidence of reading the material. However, there is little empirical research stating the 
best written means to assess that students read the materials. This study experimentally 
compared the effect of assigned reading summaries or study questions on student test 
performance. The results revealed that study questions produced higher quiz scores and 
higher preparation for the quiz, based on student feedback. Limitations of the study 
included a small sample size and extraneous activities that may have affected general 
knowledge on a topic. Results suggest that study questions focusing students on critical 
information in the required readings improve student learning. 
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INTRODUCTION 

Online instruction continues to increase in popularity with over 200 colleges and 
universities offering online instruction (National Center for Education Statistics, 2014) 
and at least 120 schools offering full online programs (National Center for Education 
Statistics, 2012). Convenience and flexibility are key benefits of online learning (Marks, 
Sibley and Arbaugh, 2005). Online learning eliminates physical barriers to education 
such as geographic distance, time constraints and household obligations (Hines and Pearl, 
2004). 

Means, Toyama, Murphy, Bakia and Jones (2009) conducted a meta-analysis that 
reported students learning online had better outcomes than those in traditional settings, 
presumably due to the delivery of content in multiple formats. Instructors may direct 
students towards publically available materials, online videos, blogs, texts or articles. 
Instructors may also create and distribute original materials including written content, 
audio-visual presentations and video lectures (Reinecke and Finn, 2015). 

Educational research has identified several pedagogical features associated with 
positive student outcomes. These include, clear behavioral objectives, accurate and 
competent models, ample opportunities for active responding, immediate feedback 
regarding the accuracy of the response with reinforcement, teaching to mastery with 
frequent direct measures that are explicitly tied to the learning objectives and using 
outcome measures to drive instructional decisions (Heward, 2006; Moran and Malott, 
2004; Skinner, 1968; Vargas, 2013). Of these, student engagement is the single best 
predictor of student learning (Gachago, Morris and Simon, 2011; Heward, 2006; Kuh, 
2005; Skinner, 1968; Vargas, 2013) therefore, it is imperative to assess student 
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engagement in online learning. Reinecke and Finn (2015) evaluated student 
engagement with materials posted in online courses. Surprisingly, their results 
demonstrated that 20 to 55% of students did not access material, handouts or audio-visual 
lectures during weekly instruction, correlating the higher percentage with students who 
received the lowest grades. 

Assessing student knowledge is one of the cornerstones of teaching. It is unclear, 
however, whether teachers know students are learning without assessing the knowledge 
through discussion, projects, tests/quizzes or similar means. The National Education 
Association (2015) suggests that instructors grade readings in order to increase student 
compliance with this task. However, the means to best assess whether the students have 
read the material has not been empirically evaluated. Hence, the purpose of this study 
was to compare the effects of two different types of evidence that students had read the 
required material based on weekly quiz scores. Students were assigned to write either a 
summary of that week’s readings or answer instructor-developed study questions. 
Comparisons were made on quiz performance and student preference. 

METHOD 

Participants 

The participants consisted of 62 graduate students enrolled in two online graduate 
courses in Applied Behavior Analysis. There were 57 female and 5 male participants 
(Table 1). The participants took these courses anywhere from their first semester in the 


program through the last semester. 
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Table 1 


Participant Make-Up Per Course 



Female 

Male 

Course A 

17 

3 

Course B 

40 

2 


Setting 

All courses were delivered entirely online and provided coursework sequence 
approved by the Behavior Analyst Certification Board (BACB®). Data were collected 
across two semesters of study, taught by three different instructors. The courses were 
delivered using the school’s learning management system, Blackboard and Canvas 
(Blackboard, 2015; Instructure, 2015). Content and activities were arranged across 11 
weekly topics. Each weekly topic contained assigned readings from either textbooks or 
readily available scholarly articles, which were posted on the course site. Additionally, 
each weekly topic contained links to audio-visual lectures, prompted forum discussions, 
instructions for weekly written assignments, specification of the reading summary, and a 
quiz. 

Dependent Measures 

The dependent variable was the mean score on the weekly quiz. Each week the 
students completed a ten-point quiz that ranged from 5-10 questions. The questions were 
based on the readings that were assigned for that week. All quiz questions were multiple 
choice or true/false. Although quizzes were automatically scored, grades were not posted 
until the quiz closed. All quizzes were timed and students had between six to ten minutes 


to complete it. 
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Experimental Design 

The experimental design was an alternating treatment design (Kazdin, 2011). 

Each section was assigned the reading summary method with alternation across sections 
(e.g., if section 1 was assigned the study questions, section 2 was assigned the reading 
summary) or the study questions. 

Independent Variable 

The independent variable was the type of assigned reading summary, either 
written responses to study questions or a narrative summary of the readings. The study 
questions contained 6-15 questions per week, and students were told to write a response 
for each question in 4-8 sentences. The reading summaries were open-ended and required 
to be 2-4 pages, double-spaced. 

The study questions were posted for students to answer, on paper, on the assigned 
week. Directions on the course site as well as syllabus were as follows: 

“Evidence of Reading Assigned Materials: For each week, you will 

submit either answers to study guides or summaries of the readings. 

You will be told which evidence of reading materials to do for each week: 

study questions or response summaries. 

A. Study Questions - on some weeks, you will be given study questions 

that highlight what the instructor considers to be the important information in 
each respective required reading. Study question responses should each be 4-6 
sentences that answer the question in your own words. Credit will not be 
given for answers taken word-for-word from the text or articles. 


B. Reading Summaries - on other weeks, you will submit a reading 
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response assignment that will be a reading summary. The reading summaries 
should be in APA format and should be comprised of a brief (2-4 pages) 
summary of the assigned readings for that week. Included in this summary 
should be a brief (half a page to one page) discussion of how you will 
incorporate the readings into your current or future setting. Credit will not be 
given for summaries taken word-for-word from the text or articles.” 


Procedure 

Each week the instructor assigned either the study questions or reading summaries 
to each course section in an alternating fashion. Students were able to complete the 
assignment at any time during the week and take the quiz after submitting the responses. 
Additionally, students were asked to complete a short survey regarding social validity 
(student preference) that included the preferred reading response method, method they 
felt produced greater learning, and method they felt prepared them best for the weekly 
quiz. 

Interobserver Agreement 

Interobserver agreement on the calculation of the mean scores was conducted on 33% of 
assignments across all four sections. Two of the professors independently calculated the 
mean score per section, per test and then compared the calculations. Agreement was 
100% accurate for calculation of mean scores. 

RESULTS 

Results showed that, on average, study questions resulted in higher quiz scores 
across both classes (see Figures 1 and 2). Course A had an overall increase of 5.84% on 
test scores with the study questions (81.36% study questions; 75.52% reading summaries) 


while Course B had a 2.59% increase of test scores with the study questions (78.24% 


31 


study questions; 75.65% reading summaries). Further analysis of the data showed that 
70% of students in Course A performed better when compared to themselves when 
assigned the study questions while 30% performed better across weeks with the reading 
summaries. Similar results were shown with Course B in that 58% of students performed 
better across weeks when assigned the study questions while only 29% of students 
performed better when assigned reading summaries. One participant performed the exact 
same on tests regardless of reading response assignment. 



123456789 10 

Week 


Figure 1. Weekly Mean Score for Reading Summaries (black bar) and Study Questions 


(Gray Bar) across Course A 
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Figure 2. Weekly Mean Score for Reading Summaries (Black Bar) and Study Questions 
(Gray Bar) Across Course B 

Social validity (student preference) data were conducted through an eight- 
question survey (see Figure 3). Participants reported spending slightly more time each 
week completing the study questions as opposed to reading summaries. Specifically, 77% 
stated spending more than two hours completing the study questions as opposed to 72% 
stating they spent more than two hours on the reading summaries. Sixty-six percent of 
participants preferred the study questions, and 83.3% reported that this method gave them 
a better “understanding of the material” or that it made them “read the material more 
carefully”. Ironically, only 66.7% stated that the study questions were the best method for 


learning. 
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Figure 3. Social Validity Data for Reading Summaries (Black Bar) and Study Questions 
(Gray Bar) Across All Students 

DISCUSSION 

The purpose of this study was to determine if there was a difference in student 
outcomes when presented with different evidence of having read the materials, that of 
study questions or reading summaries. Students completed the assigned reading response 
at any time during the week and then took the quiz. Additionally, students completed a 
short survey regarding social validity that included the preferred reading response 
method, method they felt produced greater learning, and method they felt prepared them 
best for the weekly quiz. The results of the study demonstrate that study questions 
increased both weekly and overall student performance on weekly quizzes. In addition, 
participants reported this to be the preferred method for submitting proof of reading the 
materials. 

The results demonstrated that study questions produced higher quiz scores as 
shown in 75% of the measured weeks. Factors that can be attributed to this are that the 
study questions were composed by the course instructor and highlighted the key 
components of the material, which were the basis for the weekly quizzes. Study questions 
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are similar to structured notes, a practice that has been shown to be effective in 
increasing student engagement (Heward, 2006; Vargas, 2013). 

An interesting finding was that the social validity data revealed that some students 
preferred the reading summaries, yet reported that the study questions better prepared 
them for quizzes. Most students reported that the study questions prepared them the best 
for the quiz, which aligns with the course instructor highlighting the key points in the 
study questions and, in turn, developing quiz questions from these points. 

There are several potential limitations to this study that could influence 
confidence in a causal relationship between evaluations of readings and test scores. There 
were extraneous variables that were not controlled such as forum discussion topics that 
may have influenced learning. In addition, participants’ knowledge of the content was not 
assessed prior to each unit. Some people might think that the scores varied due to 
difficulty of the material during specific weeks, however, the alternating treatments 
design would refute this claim (Kazdin, 2011). 

Future research should focus on continued ways to assess active engagement and 
immediate feedback to students while learning in an online environment (Geri, 2011; 
Grant and Spencer, 2003; Reinecke and Finn, 2015). Engagement is more than evidence 
of reading materials; engagement requires immediate feedback and ample opportunities 
to respond. Additionally, research needs to be conducted on whether exposure to course 
reading material increases learning, or if it simply promotes a better grade. Presumably, 
reading the required articles/text will increase tests scores, however, as Skinner noted, 
when testing students, “few learn at all” (Skinner, 1968, p.88). With advances in 
technology, ways to increase student engagement must be considered in online learning 


and evaluated for effectiveness and retention. 



The results of this study support previous findings that clear behavioral 
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objectives explicitly tied to the learning objectives (Heward, 2006; Moran and Malott, 
2004; Skinner, 1968; Vargas, 2013) increase student knowledge, as demonstrated through 
an increase in tests scores when participants completed the study questions. With online 
instruction advancing in popularity, there is a need to better assess how to deliver online 
instruction, as typically the delivery is based on traditional classroom teaching 
methodology (Breslow et al., 2013; Mor, Minguillon, and Carbo, 2006; Neuhauser, 

2002). It is essential that empirical methods be employed to assess learning in the 
distance format and thus move away from subjective measures of student learning that 
are commonly used in the literature today. There needs to be empirical research on the 
most effective method of delivery in online instruction and how these methods should 
differ from traditional classroom methods (Schacter and Szpunar, 2015). 
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